ja: JSONの中にJSONをネストするにはエスケープが必要になりますが、msgpackの中にmsgpackをネストして格納する場合はエスケープが必要ありません。
en: To nest JSON within JSON, escaping is required, but when nesting msgpack within msgpack, escaping is not necessary.

ja: そのため、このようなハックが有効になるようなデータを扱う場合は、msgpackをネストすることが良いプラクティスになります。
en: Therefore, when dealing with data that benefits from such a hack, nesting msgpack becomes a good practice.

ja: それでもこの機能が必要だと思うのであれば、同様の機能を実装している他のmsgpack実装を複数紹介してください。少なくとも1つはPython以外の言語用のライブラリを選んでください。
en: If you still believe this feature is necessary, please provide multiple examples of other msgpack implementations that have similar functionality. At least one of them should be a library for a language other than Python.

ja: もし他にこの機能を実装しているmsgpack実装が見つからず、それでもこの機能が本当に必要なんだと強く確信しているのであれば、その確信を私に伝えるための十分な根拠を提供してください。

en: If you cannot find any other msgpack implementations that have this feature, and you are still strongly convinced that this feature is truly necessary, please provide sufficient evidence to convince me of your conviction.

Hi! Sorry for the time it took me to reply. Here are some use cases (and below I left some msgpack packages that do similar things):

• When you have an external class (or a class from a compiled module) that implements its own msgpack serialization, it is useful (and much more efficient) to allow the class to create its own msgpack and then concatenate it into the main output (so we just bypass the class output into the main payload). That way, we avoid reimplementing the serialization logic from the class (which is error-prone: two places to update every time).
• When you receive a msgpack payload (let's say on an API) and want to just pass it along to the next service, Bypass is useful to avoid having to deserialize and re-serialize the payload (just place it as a msgpack object inside the main payload, and it's done).
• When serializing the same object multiple times (especially big objects), Bypass allows for a cache where you serialize it once and then reuse the serialized payload on the next iterations.

Note that just placing the serialized msgpack as a bin (or using an extension type for that) also increases:

1. The cost of serialization (a little bit, you need to serialize the binary header / size).
2. The size of the payload (a little bit per binary, since you need to include its header / size, but it grows with the number of embedded binaries, so it can become significant).
3. Mostly, the cost (and implementation logic) of de-serialization: you need to deserialize the main payload first, then access the object's raw binary and deserialize it, which could even be recursive, so it is very error-prone.

The Bypass class allows for flattening this object payload into the main payload, in a fast and safe (if you know it is valid msgpack) way. The resulting payload is byte-for-byte identical to serializing the object inline, so Bypass is fully transparent to the decoding side (any language / any msgpack library, it does not even need to know Bypass was used), while the raw binary (or extension type) changes the output format and forces every consumer to know the convention (possibly recursive if there are many steps).

I resorted to Bypass here because the current implementation does not provide a stream where you can write freely, but some of the example packages below do provide a stream, so it is a way to avoid the Bypass class (and another option if you want to go that way, but that would change the current default param or introduce a new one, e.g., streamed_default or something similar, which would increase implementation complexity / time).

Concatenating packb outputs externally does work, but one would need to assemble every enclosing container by hand (pack_array_header / pack_map_header, then concatenate, which could turn into O(n^2) complexity if done wrong): a Bypass value can be placed anywhere inside a Python structure (deep inside dicts and/or lists passed to packb), which manual concatenation cannot achieve without reimplementing the packer's container logic. Here are some examples of msgpack packages outside Python that allow doing something similar:

• vmihailenco/msgpack (language: Go) - has a RawMessage type that allows for bypassing the serialization of a msgpack object (here), which is written as-is to the stream by the internal Encoder.write method (here). It also exposes the underlying stream through Encoder.Writer (here), so you can also write to it directly.
• msgpack/msgpack-java (language: Java) - has a MessagePacker.writePayload method that allows for writing a msgpack object directly to the stream (here), and a zero-copy addPayload variant (here). Note that, in this case, their docs mention they are "used with" packRawStringHeader / packBinaryHeader, but that is descriptive, not a requirement: both are plain verbatim copies to the output, and their names point more to "append to the buffer" than to "pack the body of the started string/binary".
• msgpack/msgpack-c (language: C and C++) - you own the stream (it only stores a reference in C++ or a pointer in C to it), so you can write your objects directly to the stream when, for example, creating an array or a map. That one requires the container-header "solution" mentioned above (via pack_array / pack_map), but it is still a valid example of bypassing the packer logic and writing directly to the stream. They also allow for the msgpack-java workaround: msgpack_pack_str_body / msgpack_pack_v4raw_body / msgpack_pack_bin_body (in C) and packer::pack_str_body / packer::pack_v4raw_body / packer::pack_bin_body (in C++) allow for writing a msgpack object directly to the stream, but in this case their names are more explicit about what they should be used for.
• msgpack/msgpack-ruby (language: Ruby) - the packer's buffer is public (accessible through Packer#buffer), and its MessagePack::Buffer#write method appends the given bytes verbatim, with no type header (here, implemented in C).

Extra cases:

Other msgpack packages in Python

Inside Python, msgspec (which also implements msgpack) has a Raw type that does exactly this (from its docs: "Raw objects wrap pre-encoded messages. These can be added as components of larger messages without having to pay the cost of decoding and re-encoding them"). Note that Raw also appears on the decoding side, but only as a type annotation: a field annotated as Raw is kept undecoded (as a view into the original message, to be decoded later by the user); regular decoding does not produce Raw objects from Raw-encoded instances.

On CBOR

The same passthrough also exists in CBOR (a binary format very similar to msgpack), for example:

• fxamacker/cbor (language: Go) - has a RawMessage type that is written to the output with no CBOR framing added (here).
• cbor2 (language: Python) - has a CBOREncoder.write method that copies the given bytes directly to the output (here).
• QCBOR (language: C) - has a QCBOREncode_AddEncoded function ("Add some already-encoded CBOR bytes") (here).
• System.Formats.Cbor (language: C#/.NET) - has a CborWriter.WriteEncodedValue method that validates the encoded value and copies it verbatim to the output.

About the name: Raw would follow msgspec's Raw and Go's / fxamacker's RawMessage, but it could be confused with the bin format, so I avoided it. Maybe a better name for the Bypass class would be Passthrough or Flatten, but I am not sure if any of them is better. I am always open to suggestions.

Note: I have updated my branch of the repository to include the latest commits and add some basic Bypass tests.